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Abstract 



ACT's Course Placement Service uses logistic regression to model the 
relationships between outcomes in standard college courses and placement test scores. 
The logistic regression results are used together with empirical test score data to obtain 
estimates of validity statistics (e.g., proportion of correct placement decisions), given 
particular hypothetical cutoff scores. Prior research has shown that the accuracy of these 
statistics is affected by hard truncation, a condition in which no standard course outcome 
data are available for students below a certain cutoff score. Hard truncation is 
uncommon in actual placement systems, however; standard course outcome data 
typically are available for some students below the cutoff score ( soft truncation). A soft 
truncation condition of 40%, for example, is one in which 40% of the students below the 
cutoff score do not have standard course outcome data. The results of this study show 
that reasonably accurate estimates of validity statistics can be obtained even when 40%, 
and in some cases 60% or 80%, soft truncation occurs. Moreover, the slope of the logistic 
regression curve and the skewness of the test score marginal distribution have little to 
do with the relative accuracy of validity statistics unless soft truncation exceeds 40%. 
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Estimating Conditional Probabilities of Success and Other 
Course Placement Validity Statistics Under Soft Truncation 

Postsecondary institutions often use standardized test scores for deciding how to 
place students into college-level courses. Typically, students scoring at or above a 
particular cutoff score are placed into a standard course, whereas those scoring below 
the cutoff score are placed into a lower-level course, sometimes referred to as a 
"remedial" course. It is very important that correct placement decisions be made; 
otherwise, students may be adversely affected. For example, entering freshmen who are 
incorrectly placed into a standard course may be unable to complete it with a passing 
grade because they lack sufficient academic knowledge and skills. This situation could, 
of course, be disheartening for students. 

Placement decisions may also affect an institution. If institutional staff determine 
that many students need remediation, for example, then staff may attempt to schedule 
additional remedial courses. If incorrect placement decisions have been made and these 
students do not, in fact, need remediation, then scheduling efforts will have been 
needlessly undertaken. 

Sawyer (1989, 1996) applied decision theory to evaluating course placement 
systems. He considered four possible events resulting from a course placement decision: 

1. Students who score at or above the cutoff are adequately prepared for the 
standard course and are therefore successful in it (a true positive event). 

2. Students who score at or above the cutoff are, in fact, not adequately 
prepared for the standard course and are therefore not successful in it (a 
false positive event). 
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3. Students who score below the cutoff need remedial instruction and 
therefore would not have been successful in the standard course had they 
enrolled in it (a true negative event). 

4. Students who score below the cutoff do not, in fact, need remedial 
instruction and therefore would have been successful in the standard 
course had they enrolled in it (a false negative event). 

Events 3 and 4 are not directly observable in course placement systems, because 
most students who are below the cutoff score have not enrolled in the standard course 
and therefore do not have standard course grades. Consequently, the marginal 
distribution of course grades is truncated below the cutoff score. In Sawyer's (1989, 
1996) methodology, the proportions of true and false negatives (Events 3 and 4) are 
estimated by extrapolating a logistic regression function, which is estimated from the 
data of students who completed the standard course, to test scores below the cutoff 
score. Because a binary outcome variable (success, failure) is modeled as a function of 
test score, the logistic regression function yields an estimated conditional probability of 
success in the standard course: 

P [Success | X = x) = (1 + e -^ bx) Y l , (1) 

where x is a particular value of the test score X, and where a and b are estimates of the 
model parameters a and p. 

Placement system validation relies, in part, on evaluating the proportion of 
students correctly placed, given the cutoff score used for placement. The estimated 
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proportion of correct placement decisions is known as the accuracy rate (A), and is 
defined as the sum of the estimated proportions of true positives and true negatives. 

Another statistic that may be used to evaluate a placement system is the estimated 
success rate ( 5 ), which represents the proportion of students succeeding in the standard 
course, among all students who could have been placed in that course. 5 is defined as 
the proportion of true positives divided by the sum of the proportions of true and false 
positives. 

Estimated A and § are functions of the estimated conditional probabilities and the 
marginal distribution of the predictor variable (e.g., test score) in the relevant population. 
For example, the proportion of true negatives can be estimated as: 

P [ Failure , X < * 0 ]=^ 1 - P [Success \ X - x Q ] * n{x) / N 

x«„ 

for a particular cutoff score x 0 , where n(x) is the number of students with a test score of 
x, and N is the total number of students. 

Research by ACT staff indicates that truncation of course placement system data 
affects the accuracy of the estimated conditional probability of success (denoted P for 
purposes of simplification), A, and £. For example, Houston (1993) examined the effects 
of truncation on P using simulated data. Houston's findings indicate that if 25% or less 
truncation occurs (i.e., 25% or fewer students score below the cutoff and do not enroll 
in the standard course), then reasonably accurate estimates of this statistic can be 
obtained. Crouse (1996) used a bootstrap method to estimate confidence intervals for 
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validity statistics. She found that relatively large samples with less truncation yield 
relatively more accurate estimates of P, A, and optimal cutoff scores 1 . 

Schiel and Noble (1992) used a different method to investigate the effects of 
truncation on estimated validity statistics, examining actual, rather than simulated, data 
from college-level accounting, history, psychology, and biology courses. Because all 
students in the sample had enrolled in standard courses, truncation had not occurred. 
Schiel and Noble simulated the effect of truncation on these data, and found that A and 
£ are acceptably accurate when less than 15% truncation occurs. 

The preceding truncation studies were performed under conditions in which no 
standard course outcome data were available for students who were below a certain 
point (e.g., a cutoff score) in the marginal distribution of test scores. This condition is 
referred to as hard truncation. Further research examining the effects of truncation on 
estimated validity statistics is important, because hard truncation is uncommon in actual 
placement systems. Soft truncation, a condition in which standard course outcomes are 
available for some students below a particular cutoff score, is more likely to occur. It 
is conceivable that validity statistics estimated under soft truncation differ in accuracy 
from those estimated under hard truncation. 

Soft truncation can occur when cutoff scores are not strictly enforced by an 
institution and students below a particular cutoff choose to enroll in a standard course. 
There are several possible reasons for relatively low-scoring students to choose standard 

‘The estimated optimal cutoff score may be identified by examining the A at different locations in 
the marginal distribution of the predictor variable. More information about this is provided in the 
method section. 
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course enrollment. For example, they may be encouraged by an adviser to enroll in a 
remedial course, but may instead enroll in a standard course if they are confident that 
they will succeed in it. Even if students below the cutoff score are less than confident 
of succeeding, they may still enroll in a standard course, because completing a remedial 
course may delay progress toward graduation and require additional tuition and fees. 

Soft truncation can also occur when cutoff scores are used as guides for making 
placement decisions. A decision zone, which is an interval around a cutoff score, is one 
such decision-making guide. Students whose test scores are within the decision zone 
may be encouraged to provide additional information about their academic skills and 
knowledge, perhaps by taking another test. Even if the additional information indicates 
that these students should enroll in a remedial course, they may still choose to enroll in 
a standard course. 

ACT has developed the Course Placement Service (CPS) to help postsecondary 
institutions evaluate their placement systems. Among the information provided to 
institutions participating in the CPS are estimated validity statistics P, A, and 5, which 
institutional personnel can use to identify an estimated optimal cutoff score for a 
particular course. Because course placement decisions have important consequences for 
students and institutions, accurate estimates of validity statistics are imperative, 
regardless of whether they are calculated by ACT or postsecondary staff. The purpose 
of this study, therefore, was to determine the extent to which the accuracy of these 
statistics is affected by soft truncation. 
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Data 

A placement group consists of all students for whom a placement decision needs 
to be made (ACT, 1994). By definition, no truncation has occurred in the placement 
group. Data for six placement groups, each containing 500 observations previously 
simulated for use in the Crouse (1996) study, were also used in the present study. Each 
placement group consisted of the joint distribution of two random variables. One 
random variable, X, reflected the ACT Assessment score scale (1-36). The other, a binary 
variable Y, reflected a standard course outcome (success or failure). 

The simulated data were intended to be representative of data that ACT receives 
from participating CPS institutions. Two factors were varied in the simulations: the 
slope of the logistic regression curve and the skewness of the marginal distribution of 
the ACT score variable X. The simulation procedure was similar to that described in 
Houston (1993) and involved the following steps: 

1. Independently draw two random variables Xj and X 2 from two different 
gamma distributions with respective a parameters a* and b*. Forming the 
ratio X* = X, / (X 3 + X 2 ) distributes X* as a beta (a*, b*) random variable. 
Different degrees of skewness (e.g., high negative, medium negative, or 
zero) can be obtained by adjusting the parameter values a* and b*. 

2. Multiply X* by 36 and then round to the nearest positive integer. 

3. Calculate the conditional probability of success using the obtained value 
of X* and Equation 1, with a and b fixed to represent either a steep or flat 
logistic regression curve. 
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4. Select a random observation Y from a Bernoulli distribution with 
probability equal to that obtained in Step 3. 

5. Repeat Steps 1-4 500 times. 

The six simulated placement groups are described in Table 1. This table shows, 
for each placement group, the resulting skewness and logistic regression parameters a 
and b. The simulated data for placement Groups 1-3 had steep logistic regression curves 
and either high (-.62) or medium (-.29) negative skewness, or virtually zero (-.03) 
skewness, respectively. Groups 4-6, on the other hand, consisted of simulated data with 
relatively flat logistic regression curves for the three categories of skewness. 

TABLE 1 



Simulated Placement Groups 



Group 


Slope 




Skewness 




i 


Steep 


(a = -2.18, b = .11) 


High 


(-.62) 


2 


Steep 


(a = -2.46, b = .12) 


Medium 


(-.29) 


3 


Steep 


(a = -2.22, b = .11) 


Zero 


(.03) 


4 


Flat 


(a = -.79, b = .03) 


High 


(-.61) 


5 


Flat 


(a = -.26, b = .02) 


Medium 


(-.32) 


6 


Flat 


(a = -.53, b = .03) 


Zero 


(-.01) 



Figure 1 illustrates the simulated joint distribution of the ACT score (predictor) 
and course outcome variables for Placement Group 1 (steep slope, high skewness). The 
x-axis represents the simulated ACT score, and the y-axis represents the simulated 
outcome variable. Note that ACT scores are plotted in descending order from left to 




14 



8 



right. The z-axis represents the frequency of observations at a particular (x, y ) coordinate 
point. 



FIGURE 1. Joint Distribution of ACT Score 
and College Course Outcome 

(Placement Group 1 : Steep slope, high skewness) 
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It can be seen in Figure 1 that successful outcomes (Is) are associated with higher 
ACT scores. Figures A.1-A.5 in Appendix A illustrate simulated joint distributions of 
ACT scores and course outcomes for Placement Groups 2-6, respectively. 

Method 

The method used to simulate soft truncation and compare the resulting estimated 
validity statistics to those of the (non-truncated) placement group consisted of the 
following steps: 

1. Calculate estimated validity statistics for the placement group. A logistic 
regression function was fitted to the data of a particular placement group. 
The estimated conditional probabilities of success were then used together 
with the marginal distribution of the simulated ACT scale score to 
calculate estimated A and 5 at each scale score point. 

2. Simulate soft truncation. An estimated optimal cutoff score typically is the 
score that maximizes A. In addition, the maximum A corresponds to a 
conditional probability of success closest to .5. These facts were used to 
identify estimated optimal cutoff scores for placement groups and 
truncated placement groups, which will be referred to as truncation samples. 

Observations (consisting of an ( x , y) pair) below the optimal cutoff 
score were randomly selected and then deleted from each of four different 
score intervals of the ACT score distribution. The four score intervals 
chosen for each placement group were intended to be representative of 
meaningful ACT score intervals. For example, very few students earn ACT 
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scores below 8, and scores below 16 typically are considered relatively low 
for placement purposes. In addition, the intervals were constructed so that 
they contained somewhat similar percentages of observations across 
placement groups. Because optimal cutoffs varied by placement group, the 
corresponding definitions of score intervals also varied somewhat across 
placement groups. An illustration of the score intervals for Placement 
Group 1 is shown in Figure 2. 

FIGURE 2. Observations Below Optimal 
Cutoff Score, by Score Interval 
(Placement Group 1 : Steep slope, high skewness) 




Simulated ACT Assessment score 



The estimated optimal cutoff score for Placement Group 1 was 20. 
Of the 500 observations in this group, 132 were below the cutoff. Fifty- 
eight observations were in the score interval 16-19, 37 were in the score 
interval 12-15, 25 were in the score interval 8-11, and 12 were in the score 
interval 1-7. For a 20% soft truncation condition, 20% of the observations 
in each of the four intervals were randomly selected and then deleted (e.g.. 




17 



11 



7 randomly selected observations were deleted from the score interval 12- 
15). The resulting sample, reflecting 20% soft truncation, had 368 + (1 - 
.20) x 132 = 474 observations. 

This method of simulating soft truncation was intended to ensure 
that some observations remained in each score interval after truncation had 
occurred. Otherwise, it would be possible to randomly select all of the 
observations for deletion from only one score interval, resulting in an 
oddly shaped distribution (e.g., one with no observations in score interval 
12-15). Such a distribution is not likely to be encountered in an actual 
placement system. 

The particular method chosen for simulating soft truncation could 
limit the generalizability of results to some degree. It is possible to 
simulate soft truncation in different ways. For example, instead of 
retaining at least a few observations in each of several test score intervals 
below the cutoff score, soft truncation could be simulated so that no 
observations below a very low score point (e.g., ACT scale score of 7 or 
lower) were retained. It is conceivable that different methods of simulating 
soft truncation could yield somewhat different estimated validity statistics. 

An example of 80% soft truncation is illustrated in Figure 3, which 
may be compared with Figure 1 (no truncation). 
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FIGURE 3. Joint Distribution of ACT Score 
and College Course Outcome, 80% Truncated 

(Placement Group 1: Steep slope, high skewness) 




3. Repeat Step 2 500 times to obtain 500 truncation samples for a particular soft 
truncation condition (e.g., 20% soft truncation). 



O 
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4. Repeat Step 3 for different conditions of soft truncation. Truncation conditions 
of 20%, 40%, 60%, and 80% were investigated. Note that there were 4 (soft 
truncation conditions) x 500 = 2000 truncation samples. 

5. Calculate estimated P, A, and 5 at each ACT scale score point (representing 
different hypothetical cutoff scores) for each truncation sample generated in Steps 
2-4. The ACT score marginal distribution of the placement group was 
used together with the Ps calculated from truncation samples to calculate 
A and S. 

6. For each truncation condition, calculate median estimated P, A, and §, across 500 
truncation samples, by ACT score. 

7. Compare the median estimates from Step 6 to those of the placement group (Step 

1 ). 

8. Compute estimated P, A, and & at each ACT scale score point for a hard 
truncation condition (i.e., no observations below the optimal cutoff score) and 
compare these statistics to those obtained in Steps 1 and 6. 

9. Repeat the entire procedure (Steps 1-8) six times, once for each of the six 
simulated placement groups. 

This procedure yielded, for each combination of simulated placement group and 
truncation condition, estimated Ps, As, and Ss for the 36 ACT scale score points. These 
validity statistics were plotted for comparison purposes. In addition, differences 
between the validity statistics estimated from the simulated placement groups and the 
truncation samples were calculated. For example, the A for a (non-truncated) placement 



ERIC 



20 



14 



group (A n ) was subtracted from the A for a 20% soft truncation condition (A 10 ) for each 
possible ACT scale score point 2 : 



(0 



<< - 



A {i) 
A n 



A total of 36 accuracy rate differences were calculated. This also pertained to the 
calculation of A P 20 and A§ 20 . These statistics were used to evaluate the accuracy of 
estimated P, A, and S. 



Similar calculations were performed for the 40%, 60%, 80%, and hard truncation 
conditions. Mean differences were then calculated, and means of the absolute value of 
the differences were also calculated. The mean of the absolute values of the A A, for 
example, may be expressed as 



A A 



20 1 



4 

= -L T |aA 
36 tr 



©I 

20 I 



Results 

Estimated Conditional Probabilities of Success 

The effects of soft truncation on the estimated probabilities of success for 
Placement Group 1 (steep slope, high skewness) are displayed in Figure 4.1. The solid 
curve in this figure represents probabilities for the non-truncated placement group (P N ). 
Probabilities for the four soft truncation conditions and the hard truncation condition are 
shown by either dashed or dotted curves. 



2 A subscript of N will henceforth denote a non-truncated, simulated placement group (e.g., S N is 
the success rate for this group). Numerical subscripts of 20, 40, 60, or 80 will denote the four soft 
truncation conditions (e.g., P m is an estimated conditional probability of success obtained under the 
60% soft truncation condition). A subscript of H will denote the hard truncation (i.e„ 100%) condition. 
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FIGURE 4.1. Effects of Soft Truncation on 
Estimated Conditional Probability of Success 

(Placement Group 1: Steep slope, high skewness) 




ACT Assessment score 



When interpreting the results in Figure 4.1, and those throughout this study, it is 
important to remember that probabilities based on the non-truncated placement group 
data are themselves estimates and are therefore subject to error. Figure 4.1 shows that 
the effects of soft truncation on the estimated probabilities were small, generally 
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resulting in slight overestimates of this statistic. At higher scores (> 28), soft truncation 
resulted in very small underestimates of P N . The largest absolute differences between 
P N and P for any of the soft truncation conditions occurred in the lower score ranges 
(< 15). The least accurate estimates of P N occurred under the hard truncation condition. 

The results in Figure 4.2 for Placement Group 2 (steep slope, medium skewness) 
are fairly similar to those of the previous figure, illustrating that soft truncation resulted 
in slight overestimates of P N for this placement group. These overestimates occurred 
throughout the entire score range, rather than for scores of 27 or lower, as occurred for 
Placement Group 1. 
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FIGURE 4.2. Effects of Soft Truncation on 
Estimated Conditional Probability of Success 

(Placement Group 2: Steep slope, medium skewness) 




For Placement Group 3 (steep slope, zero skewness; Figure 4.3), both slight 



overestimates and underestimates of P N resulted from soft truncation. Hard truncation 



resulted in less accurate estimates of P N than those found for Placement Groups 1 and 
2. This finding may be due to the fact that a relatively large number of observations are 
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affected under hard truncation when no skewness is present in the ACT score variable. 
(The distributions for Placement Groups 1 and 2 were negatively skewed; consequently, 
there were relatively few observations in the left-hand side of these distributions that 
could be deleted under the hard truncation condition.) 

FIGURE 4.3. Effects of Soft Truncation on 
Estimated Conditional Probability of Success 

(Placement Group 3: Steep slope, zero skewness) 




ACT Assessment score 





19 



The results in Figures 4. 1-4.3 suggest that when the logistic curve was steep, soft 
truncation had very little effect on estimating probabilities of success. In addition, 
negatively skewed placement group distributions were somewhat more resistant to the 
effects of hard truncation than were non-skewed distributions, with respect to estimating 
P N . Skewness did not appear to have much effect on the accuracy of estimated P N under 
soft truncation conditions when the logistic curve was steep. 

Figure 4.4 illustrates the effects of soft truncation on Placement Group 4, which 
had a relatively flat logistic curve and high negative skewness. P N was typically 
overestimated under soft truncation. The estimates of P N calculated under the soft 
truncation conditions were, overall, somewhat less accurate than those of the preceding 
three placement groups. 
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FIGURE 4.4. Effects of Soft Truncation on 
Estimated Conditional Probability of Success 

(Placement Group 4: Flat slope, high skewness) 




ACT Assessment score 



Figure 4.5 shows results for Placement Group 5 (flat slope, medium skewness). 
The results for this group differ from those of Placement Group 4; the probability 
estimates obtained under soft truncation typically underestimated, rather than 
overestimated, P N . 
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FIGURE 4.5. Effects of Soft Truncation on 
Estimated Conditional Probability of Success 

(Placement Group 5: Flat slope, medium skewness) 




ACT Assessment score 



Results for Placement Group 6 (flat slope, zero skewness) are displayed in Figure 
4.6. Estimates of P N for this placement group were fairly accurate and comparable to 
those obtained when the logistic regression curve was relatively steep (Placement Groups 



1-3). 
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FIGURE 4.6. Effects of Soft Truncation on 
Estimated Conditional Probability of Success 

(Placement Group 6: Flat slope, zero skewness) 




ACT Assessment score 



Figures 4.1-4.6 suggest that while soft truncation did not seriously affect 
estimation accuracy, relatively less accurate estimates of P N were obtained when the 
logistic curve was flat, and the distribution of the predictor variable was moderately to 
highly skewed (i.e.. Placement Groups 4 and 5). The results in Table 2 document this 
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observation. Mean A P for each placement group, by soft truncation condition, are shown 
in this table. The column labeled "P N " shows the mean P N (over 36 scale score points) 
for each placement group. The remaining columns show mean A P and mean \AP\. For 
example, the first number beneath the column heading of "20%" (.0006) is the mean A P 
for the 20% truncation condition in Placement Group 1. This result indicates that the 
average difference between P 20 and P N , over all scale score points, was .0006. 



TABLE 2 

Effects of Soft Truncation on Estimated Probability 
of Success, by Placement Group and Truncation Condition 







Mean 




Truncation 






Placement group 


p N 


Difference 


20% 


40% 


60% 


80% 


Hard 


1: Steep slope, high skewness 


.4880 


aP 

|aP| 


.0006 

.0012 


.0016 

.0027 


.0041 

.0060 


.0097 

.0127 


.0208 

.0256 


2: Steep slope, medium skewness 


.4668 


aP 

\aP\ 


.0020 

.0020 


.0031 

.0031 


.0071 

.0071 


.0115 

.0115 


.0286 

.0292 


3: Steep slope, zero skewness 


.4566 


aP 

|aP| 


-.0009 

.0020 


-.0016 

.0035 


-.0016 

.0056 


.0008 

.0122 


.0300 

.0570 


4: Flat slope, high skewness 


.4648 


aP 

|aP| 


.0021 

.0022 


.0050 

.0051 


.0122 

.0127 


.0257 

.0272 


.0706 

.0763 


5: Flat slope, medium skewness 


.5238 


aP 

|aP| 


-.0021 

.0029 


-.0039 

.0060 


-.0092 

.0130 


-.0124 

.0181 


-.0204 

.0298 


6: Flat slope, zero skewness 


.5084 


aP 

\aP\ 


.0006 

.0012 


.0009 

.0014 


.0037 

.0064 


.0051 

.0085 


.0209 

.0328 



Results for the other three soft truncation conditions, and for hard truncation, are 
shown in the last four columns of Table 2. Note that the signs (+, -) of the A P reflect 
whether the probabilities obtained under truncation over- or underestimated P N . A 
positive value corresponds to overestimation of P N ; a negative value corresponds to 
underestimation of this statistic. 
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The results in Table 2 show that the placement groups most affected by soft 
truncation were Groups 4 (flat slope, high skewness) and 5 (flat slope, medium 
skewness). Mean \Ap\ for these groups, across all soft truncation conditions, were 
higher than those for the other four groups, ranging from .0022 (20% truncation, Group 
4) to .0272 (80% truncation, Group 4). Mean |AiP| for the other placement groups 
ranged from .0012 (20% truncation, Groups 1 and 6) to .0127 (80% truncation, Group 1). 

Soft truncation typically resulted in overestimates of P N ', this is reflected in the 
positive values of mean A P for Placement Groups 1, 2, 4, and 6. Group 4 was most 
affected by hard truncation, as indicated by its relatively large mean \Ap\ for this 
condition (.0763). 

Estimated Accuracy Rates and Optimal Cutoff Scores 

Figure 5.1 illustrates the effects of soft truncation on estimated A for Placement 
Group 1. In this figure, the maximum estimated A N corresponded to an ACT 
Assessment score of 20, indicating that this was the optimal cutoff score. The maximum 
A for the 20% and 40% soft truncation conditions were comparable, and similarly 
corresponded to a score of 20. For these soft truncation conditions, the "true” optimal 
cutoff score (20; corresponding to the maximum A N ) was therefore accurately estimated. 
The situation was somewhat different, however, for the 60% and 80% soft truncation 
conditions. The maximum A occurred at a score of 19 for both of these conditions. This 
fact is not fully discernible in Figure 5.1, but frequency distributions of As show, for 
example, that the maximum A under 60% soft truncation was .6979 at a score of 19. At 
a score of 20, this statistic was slightly smaller (.6976). The maximum A 60 and A 80 both 
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suggested that 19 was the optimal cutoff score for these respective truncation samples; 
the "true” optimal cutoff score was therefore underestimated by one ACT scale score 
point under 60% and 80% soft truncation. 



FIGURE 5.1 . Effects of Soft Truncation on 
Estimated Accuracy Rate 

(Placement Group 1 : Steep slope, high skewness) 
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Figures 5.2-5.6 show the effects of soft truncation on estimated A for the 
remaining five placement groups. In general, these figures indicate that relatively more 
accurate estimates of optimal cutoff scores were obtained when the logistic regression 
curve for a particular truncation sample was fairly steep. If, on the other hand, the 
logistic curve was relatively flat, then truncation samples with virtually no skewness of 
the ACT score yielded the most accurate estimates of optimal cutoff scores, regardless 
of the extent of soft truncation. 
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FIGURE 5.2. Effects of Soft Truncation on 
Estimated Accuracy Rate 

(Placement Group 2: Steep slope, medium skewness) 
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FIGURE 5.3. Effects of Soft Truncation on 
Estimated Accuracy Rate 

(Placement Group 3: Steep slope, zero skewness) 
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FIGURE 5.4. Effects of Soft Truncation on 
Estimated Accuracy Rate 

(Placement Group 4: Flat slope, high skewness) 
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FIGURE 5.5. Effects of Soft Truncation on 
Estimated Accuracy Rate 

(Placement Group 5: Flat slope, medium skewness) 
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FIGURE 5.6. Effects of Soft Truncation on 
Estimated Accuracy Rate 

(Placement Group 6: Flat slope, zero skewness) 
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The maximum A obtained under hard truncation yielded underestimates of the 
optimal cutoff score for all placement groups. In some cases, the underestimation was 
extreme. For example, in Figure 5.4, A N was maximized at a score of 23, indicating that 
this was the true" optimal cutoff score. In comparison, A H was maximized at a score 
of 6. 

Mean A A and |aA| are reported, by placement group, in Table 3. Relatively 
accurate estimates of A N were found for Placement Groups 1 (steep slope, high 
skewness) and 6 (flat slope, zero skewness) across all soft truncation conditions. |AA | 
ranged from .0004 (Group 1; 20% truncation) to .0046 (Group 1; 80% truncation) for these 
groups. Relatively less accurate estimates of A N were found for Placement Groups 2 
(steep slope, medium skewness), 4 (flat slope, high skewness), and 5 (flat slope, medium 
skewness); \aA\ ranged from .0009 (Group 4, 20% truncation) to .0101 (Group 4, 80% 
truncation) for these groups. Note that mean AS and |a 5| are also reported in Table 
3; these statistics will be examined in the following section on estimated success rates. 
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TABLE 3 

Effects of Soft Truncation on Estimated Accuracy Rate 
and Success Rate, by Placement Group and Truncation Condition 



Placement group 






Mean 






Truncation 










Difference 


20% 


40% 


60% 


80% 


Hard 


1: Steep slope, high skewness 


.6225 


.7241 


aA 


-.0004 


-.0008 


-.0016 


-.0028 


-.0047 








Ml 


.0004 


.0010 


.0021 


.0046 


.0096 








AS 


-.0005 


-.0008 


-.0012 


-.0012 


-.0003 








Mi 


.0005 


.0010 


.0018 


.0038 


.0073 


2: Steep slope, medium skewness 


.5822 


.6625 


AA 


.0006 


.0012 


.0014 


.0015 


-.0011 








Ml 


.0017 


.0030 


.0057 


.0090 


.0206 








AS 


.0024 


.0046 


.0075 


.0107 


.0164 








Ml 


.0024 


.0046 


.0075 


.0107 


.0169 


3: Steep slope, zero skewness 


.5597 


.5738 


AA 


-.0006 


-.0011 


-.0017 


-.0038 


-.0155 








Ml 


.0011 


.0017 


.0021 


.0047 


.0349 








AS 


-.0022 


-.0038 


-.0050 


-.0061 


.0028 








Ml 


.0022 


.0038 


.0050 


.0073 


.0291 


4: Flat slope, high skewness 


.5307 


.5495 


aA 


.0000 


.0002 


.0001 


-.0007 


-.0036 








Ml 


.0009 


.0022 


.0051 


.0101 


.0268 








AS 


.0007 


.0018 


.0036 


.0059 


.0127 








Ml 


.0008 


.0020 


.0043 


.0082 


.0218 


5: Flat slope, medium skewness 


.5212 


.5611 


A A 


.0005 


.0013 


.0026 


.0037 


.0061 








Ml 


.0013 


.0021 


.0053 


.0070 


.0118 








aS 


-.0001 


.0008 


.0005 


.0013 


.0018 








Ml 


.0011 


.0020 


.0045 


.0061 


.0105 


6: Flat slope, zero skewness 


.5197 


.5530 


aA 


-.0002 


-.0003 


-.0015 


-.0019 


-.0075 








Ml 


.0006 


‘.0008 


.0033 


.0045 


.0173 








AS 


.0000 


.0002 


.0001 


.0003 


.0017 








Ml 


.0007 


.0006 


.0029 


.0040 


.0144 



Even though estimates of A N were relatively inaccurate for Placement Group 2, 



the corresponding estimated optimal cutoff score was acceptably accurate across soft 
truncation conditions (see Figure 5.2). This was not the case, however, for Placement 
Groups 4 and 5. Although the inaccuracy of optimal cutoff score estimates for these 



placement groups is illustrated in Figures 5.4 and 5.5, a clearer illustration is provided 



in Table 4. 




40 



34 



The effect of truncation on the estimation of optimal cutoff scores is summarized 
in Table 4. For each placement group, this table shows the estimated optimal cutoff 
score (corresponding to the maximum A) at each truncation condition. The estimated 
P and § are also shown. For Placement Group 1, for example, the value of A was 
maximized at a score of 20 when no truncation was present. The maximum A N was 
.70079; P N and P N were .52694 and .72336, respectively. Under the 20% soft truncation 
condition, the optimal cutoff score was again estimated as 20 for Group 1. The 
corresponding maximum A 20 was .69996 (recall that this statistic is a median calculated 
across 500 truncation samples). When soft truncation was at 60%, however, a cutoff of 
19 was incorrectly estimated as optimal. Similar findings occurred under 80% soft 
truncation and under hard truncation. 
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TABLE 4 



How Truncation Affects the Estimation 
of Optimal Cutoff Scores, by Placement Group 



Placement group 


Truncation 


Optimal 
cutoff score 


p 


Max(A) 


5 


1: Steep slope, high skewness 


None 1 


20 


.52694 


.70079 


.72336 




20% 


20 


.52717 


.69996 


.72277 




40% 


20 


.52850 


.69918 


.72235 




60% 


19 


.50360 


.69787 


.71448 




80% 


19 


.51073 


.69553 


.71453 




Hard 


19 


.52384 


.69192 


.71604 


2: Steep slope, medium skewness 


None 


20 


.50027 


.66771 


.66753 




20% 


20 


.50280 


.66862 


.67006 




40% 


20 


.50479 


.66962 


.67250 




60% 


20 


.50994 


.66952 


.67564 




80% 


20 


.51586 


.66879 


.67901 




Hard 


19 


.50781 


.66076 


.67338 


3: Steep slope, zero skewness 


None 


21 


.51100 


.64808 


.61274 




20% 


21 


.50886 


.64674 


.61037 




40% 


21 


.50806 


.64629 


.60780 




60% 


21 


.50721 


.64456 


.60604 




80% 


21 


.50876 


.63858 


.60313 




Hard 


20 


.51309 


.60351 


.59455 


4: Flat slope, high skewness 


None 


23 


.50284 


.55357 


.55422 




20% 


23 


.50458 


.55316 


.55528 




40% 


23 


.50660 


.55244 


.55603 




60% 


22 


.50368 


.55099 


.55393 




80% 


20 


.50068 


.54801 


.55244 




Hard 


6 


.50149 


.55237 


.55276 


5: Flat slope, medium skewness 


None 


14 


.50236 


.54379 


.54845 




20% 


15 


.50403 


.54399 


.54958 




40% 


15 


.50124 


.54466 


.54916 




60% 


16 


.49961 


.54572 


.55046 




80% 


17 


.50196 


.54739 


.55304 




Hard 


19 


.50737 


.55158 


.55854 


6: Flat slope, zero skewness 


None 


18 


.50475 


.54281 


.54632 




20% 


18 


.50556 


.54224 


.54640 




40% 


18 


.50560 


.54216 


.54669 




60% 


18 


.50888 


.53951 


.54694 




80% 


17 


.50360 


.53911 


.54471 




Hard 


12 


.50133 


.53478 


. .53842 



'Placement group (non-truncated). 
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The largest difference between the optimal cutoff score estimated for a (non- 
truncated) placement group and one estimated for any soft truncation condition occurred 
for Placement Groups 4 (flat slope, high skewness) and 5 (flat slope, medium skewness). 
For Placement Group 4, 80% soft truncation yielded a three-point underestimate of the 
optimal cutoff score (23 vs. 20). For Placement Group 5, this same soft truncation 
condition yielded a three-point overestimate of the optimal cutoff score (14 vs. 17). The 
most accurate optimal cutoff score estimates occurred for Placement Groups 2 and 3, 
regardless of the extent of soft truncation. 

Estimated Success Rates 

Mean A§ and mean | A5 1 are reported in Table 3. The most accurate estimates 
of § N occurred for Placement Groups 1 (steep slope, high skewness) and 6 (flat slope, 
zero skewness). |AS| ranged from .0005 (Group 1, 20% truncation) to .0040 (Group 6, 
80% truncation) for these groups. Relatively less accurate estimates of were found, 
across all soft truncation conditions, for Placement Group 2 (steep slope, medium 
skewness). |A$| ranged from .0024 (20% truncation) to .0107 (80% truncation) for this 
group. The effect of soft truncation on estimated 5 is displayed graphically, by 
placement group, in Figures B.1-B.6 in Appendix B. 

Discussion 

Given a particular joint distribution of college course grades and placement test 
scores, soft truncation is likely to have very little effect on estimating conditional 
probabilities of success if the estimated logistic regression curve is steep. If, on the other 
hand, the logistic regression curve is relatively flat, then samples with no skewness in 
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the marginal distribution of the placement test score will provide the most accurate 
estimates of these statistics when soft truncation is present. 

Under soft truncation, optimal ACT Assessment cutoff scores will likely be 
underestimated (i.e., will be lower than the optimal cutoff score estimated when no 
truncation is present) by a maximum of one scale score point when a logistic regression 
curve is fairly steep. In the case of relatively flat logistic curves, optimal ACT 
Assessment cutoff scores may be under- or overestimated by a maximum of three scale 
score points. Accurate estimates of the conditional probability of success, accuracy rate, 
and optimal ACT Assessment cutoff score can likely be obtained even when 40%, and 
in some cases 60% or 80%, soft truncation occurs. Moreover, the slope of the logistic 
regression curve and the skewness of the test score marginal distribution have little to 
do with the relative accuracy of these statistics unless soft truncation exceeds 40%. 

The findings of this study have implications for the estimated validity statistics 
that postsecondary institutions use to establish cutoff scores for course placement. An 
optimal cutoff score for a particular college-level standard course that is underestimated 
by one ACT Assessment scale score point may have little practical consequence for an 
institution and its students. For example, it is likely that relatively few students who 
otherwise would have been placed into a remedial course would be placed (incorrectly) 
into a corresponding standard course if the optimal ACT Assessment cutoff score were 
to be underestimated by one scale score point. In comparison, if an optimal cutoff were 
to be underestimated by three ACT Assessment scale score points, relatively more 
students would be affected. 
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When interpreting estimated validity statistics, it is important for postsecondary 
staff to evaluate course placement data with respect to skewness of the test score 
distribution and slope of the logistic regression curve. For example, if it were found that 
the logistic regression curve was relatively flat for a particular institution's data, that a 
moderate degree of negative skewness was present in the marginal distribution of the 
ACT Assessment score, and that 80% of the students who scored below the institution's 
present cutoff score had not enrolled in and completed the standard course, then it is 
possible that the optimal cutoff score could be underestimated by as many as three scale 
score points. It may be possible to direct future research efforts on truncation toward 
developing some type of correction for validity statistics estimated under the conditions 
described in this example. Suitable corrections would likely permit postsecondary 
institutions to derive greater benefit from using estimated validity statistics to evaluate 
their placement systems and establish cutoff scores. 
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Appendix A 

Joint Distributions of ACT Score and 
College Course Outcome for Placement Groups 2-5 
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FIGURE A.1. Joint Distribution of ACT Score 
and College Course Outcome 

(Placement Group 2: Steep slope, medium skewness) 
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FIGURE A.2. Joint Distribution of ACT Score 
and College Course Outcome 

(Placement Group 3: Steep slope, zero skewness) 
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FIGURE A.3. Joint Distribution of ACT Score 
and College Course Outcome 

(Placement Group 4: Flat slope, high skewness) 
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FIGURE A.4. Joint Distribution of ACT Score 
and College Course Outcome 

(Placement Group 5: Flat slope, medium skewness) 
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FIGURE A.5. Joint Distribution of ACT Score 
and College Course Outcome 

(Placement Group 6: Flat slope, zero skewness) 
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Appendix B 

Effects of Soft Truncation on 
Estimated Success Rate, by Placement Group 
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FIGURE B.l. Effects of Soft Truncation on 
Estimated Success Rate 

(Placement Group 1: Steep slope, high skewness) 
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FIGURE B.2. Effects of Soft Truncation on 
Estimated Success Rate 

(Placement Group 2: Steep slope, medium skewness) 
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FIGURE B.3. Effects of Soft Truncation on 
Estimated Success Rate 

(Placement Group 3: Steep slope, zero skewness) 
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FIGURE B.4. Effects of Soft Truncation on 
Estimated Success Rate 

(Placement Group 4: Flat slope, high skewness) 
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FIGURE B.5. Effects of Soft Truncation on 
Estimated Success Rate 

(Placement Group 5: Flat slope, medium skewness) 
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FIGURE B.6. Effects of Soft Truncation on 
Estimated Success Rate 

(Placement Group 6: Flat slope, zero skewness) 
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